Add integration tests project #1155

jbesraa · 2024-09-04T09:50:33Z

Adds an initial integration tests project to the roles workspace.

This crate should hold tests related to the other crates in the roles workspace, allowing us to write integration tests for those.

github-actions · 2024-09-04T09:54:49Z

Bencher

Report	Wed, September 4, 2024 at 13:00:20 UTC
Project	Stratum V2 (SRI)
Branch	2024-09-04-int-test
Testbed	sv1

Click to view all benchmark results

Benchmark	Estimated Cycles	Estimated Cycles Results estimated cycles \| (Δ%)	Estimated Cycles Lower Boundary estimated cycles \| (%)	Estimated Cycles Upper Boundary estimated cycles \| (%)	Instructions	Instructions Results instructions \| (Δ%)	Instructions Lower Boundary instructions \| (%)	Instructions Upper Boundary instructions \| (%)	L1 Accesses	L1 Accesses Results accesses \| (Δ%)	L1 Accesses Lower Boundary accesses \| (%)	L1 Accesses Upper Boundary accesses \| (%)	L2 Accesses	L2 Accesses Results accesses \| (Δ%)	L2 Accesses Lower Boundary accesses \| (%)	L2 Accesses Upper Boundary accesses \| (%)	RAM Accesses	RAM Accesses Results accesses \| (Δ%)	RAM Accesses Lower Boundary accesses \| (%)	RAM Accesses Upper Boundary accesses \| (%)
get_authorize	✅ (view plot)	8,481.00 (+0.45%)	8,166.98 (96.30%)	8,719.82 (97.26%)	✅ (view plot)	3,772.00 (+0.64%)	3,640.31 (96.51%)	3,855.91 (97.82%)	✅ (view plot)	5,296.00 (+0.74%)	5,108.91 (96.47%)	5,405.23 (97.98%)	✅ (view plot)	7.00 (-4.51%)	4.29 (61.35%)	10.37 (67.53%)	✅ (view plot)	90.00 (+0.01%)	85.99 (95.54%)	93.99 (95.75%)
get_submit	✅ (view plot)	95,378.00 (-0.12%)	94,927.59 (99.53%)	96,066.62 (99.28%)	✅ (view plot)	59,522.00 (+0.08%)	59,209.93 (99.48%)	59,736.13 (99.64%)	✅ (view plot)	85,508.00 (+0.10%)	85,047.50 (99.46%)	85,789.45 (99.67%)	✅ (view plot)	49.00 (-1.13%)	33.37 (68.11%)	65.75 (74.53%)	✅ (view plot)	275.00 (-2.09%)	273.33 (99.39%)	288.43 (95.34%)
get_subscribe	✅ (view plot)	8,000.00 (-0.03%)	7,722.28 (96.53%)	8,282.89 (96.58%)	✅ (view plot)	2,848.00 (+0.44%)	2,732.85 (95.96%)	2,938.30 (96.93%)	✅ (view plot)	3,985.00 (+0.54%)	3,826.32 (96.02%)	4,101.24 (97.17%)	✅ (view plot)	12.00 (-11.02%)	5.97 (49.78%)	21.00 (57.15%)	✅ (view plot)	113.00 (-0.41%)	108.88 (96.36%)	118.05 (95.72%)
serialize_authorize	✅ (view plot)	12,264.00 (+0.15%)	11,945.40 (97.40%)	12,546.05 (97.75%)	✅ (view plot)	5,343.00 (+0.45%)	5,211.31 (97.54%)	5,426.91 (98.45%)	✅ (view plot)	7,459.00 (+0.54%)	7,270.91 (97.48%)	7,567.69 (98.56%)	✅ (view plot)	9.00 (-5.85%)	5.57 (61.91%)	13.55 (66.43%)	✅ (view plot)	136.00 (-0.39%)	131.38 (96.60%)	141.69 (95.99%)
serialize_deserialize_authorize	✅ (view plot)	24,867.00 (+1.13%)	24,183.63 (97.25%)	24,993.27 (99.49%)	✅ (view plot)	9,920.00 (+0.21%)	9,774.91 (98.54%)	10,024.36 (98.96%)	✅ (view plot)	14,017.00 (+0.32%)	13,788.41 (98.37%)	14,155.46 (99.02%)	✅ (view plot)	35.00 (-2.13%)	29.44 (84.11%)	42.09 (83.16%)	✅ (view plot)	305.00 (+2.27%)	288.09 (94.46%)	308.35 (98.91%)
serialize_deserialize_handle_authorize	✅ (view plot)	30,367.00 (+0.43%)	29,901.05 (98.47%)	30,572.00 (99.33%)	✅ (view plot)	12,097.00 (+0.04%)	11,982.54 (99.05%)	12,202.21 (99.14%)	✅ (view plot)	17,142.00 (+0.13%)	16,965.57 (98.97%)	17,273.17 (99.24%)	✅ (view plot)	55.00 (-3.83%)	49.55 (90.10%)	64.83 (84.83%)	✅ (view plot)	370.00 (+0.93%)	357.63 (96.66%)	375.58 (98.51%)
serialize_deserialize_handle_submit	✅ (view plot)	126,463.00 (+0.01%)	125,877.37 (99.54%)	127,024.85 (99.56%)	✅ (view plot)	73,363.00 (+0.12%)	72,952.79 (99.44%)	73,594.75 (99.69%)	✅ (view plot)	105,198.00 (+0.15%)	104,575.16 (99.41%)	105,502.75 (99.71%)	✅ (view plot)	109.00 (-2.93%)	89.77 (82.36%)	134.81 (80.85%)	✅ (view plot)	592.00 (-0.63%)	589.05 (99.50%)	602.42 (98.27%)
serialize_deserialize_handle_subscribe	✅ (view plot)	27,907.00 (+0.92%)	27,041.51 (96.90%)	28,264.90 (98.73%)	✅ (view plot)	9,666.00 (+0.24%)	9,536.54 (98.66%)	9,750.07 (99.14%)	✅ (view plot)	13,682.00 (+0.30%)	13,492.09 (98.61%)	13,788.99 (99.22%)	✅ (view plot)	59.00 (-8.17%)	55.18 (93.53%)	73.31 (80.48%)	✅ (view plot)	398.00 (+1.74%)	375.13 (94.25%)	407.24 (97.73%)
serialize_deserialize_submit	✅ (view plot)	115,445.00 (+0.24%)	114,601.45 (99.27%)	115,745.54 (99.74%)	✅ (view plot)	68,223.00 (+0.20%)	67,775.42 (99.34%)	68,400.31 (99.74%)	✅ (view plot)	97,935.00 (+0.23%)	97,237.87 (99.29%)	98,174.44 (99.76%)	✅ (view plot)	65.00 (-0.21%)	52.63 (80.97%)	77.65 (83.71%)	✅ (view plot)	491.00 (+0.25%)	483.35 (98.44%)	496.17 (98.96%)
serialize_deserialize_subscribe	✅ (view plot)	23,405.00 (+1.40%)	22,464.03 (95.98%)	23,698.52 (98.76%)	✅ (view plot)	8,225.00 (+0.33%)	8,090.06 (98.36%)	8,306.06 (99.02%)	✅ (view plot)	11,590.00 (+0.37%)	11,398.85 (98.35%)	11,696.72 (99.09%)	✅ (view plot)	39.00 (+1.97%)	32.97 (84.54%)	43.52 (89.60%)	✅ (view plot)	332.00 (+2.45%)	308.46 (92.91%)	339.67 (97.74%)
serialize_submit	✅ (view plot)	99,771.00 (-0.11%)	99,350.27 (99.58%)	100,420.03 (99.35%)	✅ (view plot)	61,566.00 (+0.08%)	61,250.58 (99.49%)	61,784.02 (99.65%)	✅ (view plot)	88,351.00 (+0.10%)	87,885.22 (99.47%)	88,640.12 (99.67%)	✅ (view plot)	51.00 (+0.43%)	36.77 (72.10%)	64.79 (78.72%)	✅ (view plot)	319.00 (-1.79%)	317.87 (99.64%)	331.77 (96.15%)
serialize_subscribe	✅ (view plot)	11,341.00 (-0.31%)	11,078.69 (97.69%)	11,674.58 (97.14%)	✅ (view plot)	4,195.00 (+0.30%)	4,079.85 (97.26%)	4,285.30 (97.89%)	✅ (view plot)	5,841.00 (+0.35%)	5,683.20 (97.30%)	5,958.14 (98.03%)	✅ (view plot)	15.00 (+4.34%)	8.75 (58.31%)	20.01 (74.98%)	✅ (view plot)	155.00 (-1.08%)	150.84 (97.32%)	162.53 (95.37%)

Bencher - Continuous Benchmarking
View Public Perf Page
Docs | Repo | Chat | Help

github-actions · 2024-09-04T09:54:49Z

Bencher

Report	Wed, September 11, 2024 at 08:23:57 UTC
Project	Stratum V2 (SRI)
Branch	1155/merge
Testbed	sv1

Click to view all benchmark results

Benchmark	Latency	Latency Results nanoseconds (ns) \| (Δ%)	Latency Upper Boundary nanoseconds (ns) \| (%)
client-submit-serialize	✅ (view plot)	6,941.60 (+1.65%)	7,567.31 (91.73%)
client-submit-serialize-deserialize	✅ (view plot)	7,834.80 (-0.02%)	8,772.66 (89.31%)
client-submit-serialize-deserialize-handle/client-submit-serialize-deserialize-handle	✅ (view plot)	8,438.60 (+0.30%)	9,233.22 (91.39%)
client-sv1-authorize-serialize-deserialize-handle/client-sv1-authorize-serialize-deserialize-handle	✅ (view plot)	903.68 (-1.27%)	983.70 (91.87%)
client-sv1-authorize-serialize-deserialize/client-sv1-authorize-serialize-deserialize	✅ (view plot)	723.43 (+1.03%)	766.00 (94.44%)
client-sv1-authorize-serialize/client-sv1-authorize-serialize	✅ (view plot)	246.54 (-1.35%)	263.15 (93.69%)
client-sv1-get-authorize/client-sv1-get-authorize	✅ (view plot)	156.19 (-1.02%)	166.52 (93.80%)
client-sv1-get-submit	✅ (view plot)	6,544.50 (-0.96%)	7,292.59 (89.74%)
client-sv1-get-subscribe/client-sv1-get-subscribe	✅ (view plot)	275.28 (-0.27%)	300.75 (91.53%)
client-sv1-subscribe-serialize-deserialize-handle/client-sv1-subscribe-serialize-deserialize-handle	✅ (view plot)	779.02 (+1.51%)	830.85 (93.76%)
client-sv1-subscribe-serialize-deserialize/client-sv1-subscribe-serialize-deserialize	✅ (view plot)	644.30 (+2.59%)	683.51 (94.26%)
client-sv1-subscribe-serialize/client-sv1-subscribe-serialize	✅ (view plot)	202.83 (-2.79%)	226.78 (89.44%)

Bencher - Continuous Benchmarking
View Public Perf Page
Docs | Repo | Chat | Help

github-actions · 2024-09-04T09:54:51Z

Bencher

Report	Wed, September 4, 2024 at 13:00:22 UTC
Project	Stratum V2 (SRI)
Branch	2024-09-04-int-test
Testbed	sv2

🚨 1 ALERT: Threshold Boundary Limit exceeded!

Benchmark	Measure (units)	View	Value	Lower Boundary	Upper Boundary
client_sv2_handle_message_mining	Latency (nanoseconds (ns))	🚨 (view plot \| view alert)	90.26 (+21.59%)	63.29 (70.12%)	85.17 (105.97%)

Click to view all benchmark results

Benchmark	Latency	Latency Results nanoseconds (ns) \| (Δ%)	Latency Lower Boundary nanoseconds (ns) \| (%)	Latency Upper Boundary nanoseconds (ns) \| (%)
client_sv2_handle_message_common	✅ (view plot)	44.42 (-0.54%)	43.33 (97.55%)	45.99 (96.59%)
client_sv2_handle_message_mining	🚨 (view plot \| view alert)	90.26 (+21.59%)	63.29 (70.12%)	85.17 (105.97%)
client_sv2_mining_message_submit_standard	✅ (view plot)	14.65 (-0.08%)	14.59 (99.59%)	14.73 (99.44%)
client_sv2_mining_message_submit_standard_serialize	✅ (view plot)	271.76 (+1.97%)	246.53 (90.72%)	286.51 (94.85%)
client_sv2_mining_message_submit_standard_serialize_deserialize	✅ (view plot)	600.88 (+1.14%)	557.37 (92.76%)	630.89 (95.24%)
client_sv2_open_channel	✅ (view plot)	168.00 (+1.16%)	159.33 (94.84%)	172.80 (97.22%)
client_sv2_open_channel_serialize	✅ (view plot)	284.49 (+1.30%)	264.69 (93.04%)	296.99 (95.79%)
client_sv2_open_channel_serialize_deserialize	✅ (view plot)	375.01 (-0.62%)	325.71 (86.85%)	428.97 (87.42%)
client_sv2_setup_connection	✅ (view plot)	160.78 (-1.44%)	149.93 (93.25%)	176.34 (91.18%)
client_sv2_setup_connection_serialize	✅ (view plot)	432.24 (-7.65%)	410.10 (94.88%)	525.95 (82.18%)
client_sv2_setup_connection_serialize_deserialize	✅ (view plot)	955.74 (-2.85%)	884.86 (92.58%)	1,082.78 (88.27%)

Bencher - Continuous Benchmarking
View Public Perf Page
Docs | Repo | Chat | Help

github-actions · 2024-09-04T09:54:54Z

Bencher

Report	Wed, September 4, 2024 at 13:00:25 UTC
Project	Stratum V2 (SRI)
Branch	2024-09-04-int-test
Testbed	sv2

Click to view all benchmark results

Benchmark	Estimated Cycles	Estimated Cycles Results estimated cycles \| (Δ%)	Estimated Cycles Lower Boundary estimated cycles \| (%)	Estimated Cycles Upper Boundary estimated cycles \| (%)	Instructions	Instructions Results instructions \| (Δ%)	Instructions Lower Boundary instructions \| (%)	Instructions Upper Boundary instructions \| (%)	L1 Accesses	L1 Accesses Results accesses \| (Δ%)	L1 Accesses Lower Boundary accesses \| (%)	L1 Accesses Upper Boundary accesses \| (%)	L2 Accesses	L2 Accesses Results accesses \| (Δ%)	L2 Accesses Lower Boundary accesses \| (%)	L2 Accesses Upper Boundary accesses \| (%)	RAM Accesses	RAM Accesses Results accesses \| (Δ%)	RAM Accesses Lower Boundary accesses \| (%)	RAM Accesses Upper Boundary accesses \| (%)
client_sv2_handle_message_common	✅ (view plot)	2,125.00 (+2.53%)	1,974.34 (92.91%)	2,170.90 (97.89%)	✅ (view plot)	473.00 (+0.31%)	457.98 (96.82%)	485.12 (97.50%)	✅ (view plot)	735.00 (+0.23%)	712.81 (96.98%)	753.85 (97.50%)	✅ (view plot)	5.00 (-18.71%)	-0.39 (-7.72%)	12.69 (39.41%)	✅ (view plot)	39.00 (+4.31%)	34.64 (88.82%)	40.13 (97.18%)
client_sv2_handle_message_mining	✅ (view plot)	8,146.00 (-0.64%)	8,075.17 (99.13%)	8,321.68 (97.89%)	✅ (view plot)	2,137.00 (+0.20%)	2,094.80 (98.03%)	2,170.63 (98.45%)	✅ (view plot)	3,166.00 (+0.39%)	3,092.10 (97.67%)	3,215.04 (98.47%)	✅ (view plot)	30.00 (-17.66%)	29.03 (96.78%)	43.83 (68.44%)	✅ (view plot)	138.00 (-0.67%)	136.18 (98.68%)	141.69 (97.39%)
client_sv2_mining_message_submit_standard	✅ (view plot)	6,277.00 (-0.04%)	6,173.01 (98.34%)	6,385.50 (98.30%)	✅ (view plot)	1,750.00 (-0.01%)	1,737.31 (99.27%)	1,762.99 (99.26%)	✅ (view plot)	2,552.00 (-0.06%)	2,533.07 (99.26%)	2,573.74 (99.16%)	✅ (view plot)	17.00 (-1.37%)	11.34 (66.68%)	23.14 (73.48%)	✅ (view plot)	104.00 (+0.01%)	101.10 (97.21%)	106.88 (97.30%)
client_sv2_mining_message_submit_standard_serialize	✅ (view plot)	14,666.00 (-0.63%)	14,523.62 (99.03%)	14,993.57 (97.82%)	✅ (view plot)	4,694.00 (-0.00%)	4,681.31 (99.73%)	4,706.99 (99.72%)	✅ (view plot)	6,761.00 (+0.06%)	6,737.17 (99.65%)	6,776.15 (99.78%)	✅ (view plot)	41.00 (-8.35%)	36.37 (88.71%)	53.10 (77.21%)	✅ (view plot)	220.00 (-1.01%)	215.68 (98.03%)	228.79 (96.16%)
client_sv2_mining_message_submit_standard_serialize_deserialize	✅ (view plot)	27,524.00 (+0.05%)	27,196.49 (98.81%)	27,822.07 (98.93%)	✅ (view plot)	10,585.00 (+0.21%)	10,508.25 (99.27%)	10,617.53 (99.69%)	✅ (view plot)	15,404.00 (+0.22%)	15,290.65 (99.26%)	15,449.80 (99.70%)	✅ (view plot)	79.00 (-3.37%)	74.74 (94.61%)	88.77 (89.00%)	✅ (view plot)	335.00 (-0.05%)	326.43 (97.44%)	343.87 (97.42%)
client_sv2_open_channel	✅ (view plot)	4,405.00 (-0.70%)	4,254.88 (96.59%)	4,617.61 (95.40%)	✅ (view plot)	1,461.00 (+0.05%)	1,447.94 (99.11%)	1,472.66 (99.21%)	✅ (view plot)	2,160.00 (+0.19%)	2,136.21 (98.90%)	2,175.43 (99.29%)	✅ (view plot)	8.00 (-23.05%)	4.46 (55.75%)	16.33 (48.98%)	✅ (view plot)	63.00 (-1.05%)	58.82 (93.36%)	68.52 (91.94%)
client_sv2_open_channel_serialize	✅ (view plot)	13,992.00 (-1.00%)	13,828.97 (98.83%)	14,436.88 (96.92%)	✅ (view plot)	5,064.00 (+0.01%)	5,050.94 (99.74%)	5,075.66 (99.77%)	✅ (view plot)	7,332.00 (+0.13%)	7,300.08 (99.56%)	7,344.45 (99.83%)	✅ (view plot)	30.00 (-14.38%)	28.15 (93.82%)	41.93 (71.55%)	✅ (view plot)	186.00 (-1.89%)	181.16 (97.40%)	198.01 (93.93%)
client_sv2_open_channel_serialize_deserialize	✅ (view plot)	22,623.00 (-0.09%)	22,326.91 (98.69%)	22,959.34 (98.54%)	✅ (view plot)	8,027.00 (+0.29%)	7,950.62 (99.05%)	8,057.46 (99.62%)	✅ (view plot)	11,678.00 (+0.29%)	11,565.05 (99.03%)	11,723.17 (99.61%)	✅ (view plot)	75.00 (+2.01%)	64.54 (86.05%)	82.50 (90.91%)	✅ (view plot)	302.00 (-0.58%)	294.82 (97.62%)	312.69 (96.58%)
client_sv2_setup_connection	✅ (view plot)	4,703.00 (+0.29%)	4,616.83 (98.17%)	4,762.30 (98.75%)	✅ (view plot)	1,502.00 (+0.05%)	1,488.94 (99.13%)	1,513.66 (99.23%)	✅ (view plot)	2,278.00 (+0.05%)	2,257.21 (99.09%)	2,296.64 (99.19%)	✅ (view plot)	9.00 (-3.93%)	4.25 (47.27%)	14.48 (62.15%)	✅ (view plot)	68.00 (+0.60%)	65.52 (96.36%)	69.67 (97.61%)
client_sv2_setup_connection_serialize	✅ (view plot)	16,170.00 (-0.26%)	15,974.65 (98.79%)	16,451.01 (98.29%)	✅ (view plot)	5,963.00 (+0.01%)	5,949.94 (99.78%)	5,974.66 (99.80%)	✅ (view plot)	8,665.00 (+0.07%)	8,635.98 (99.67%)	8,682.41 (99.80%)	✅ (view plot)	38.00 (-9.20%)	31.35 (82.49%)	52.35 (72.58%)	✅ (view plot)	209.00 (-0.40%)	203.50 (97.37%)	216.17 (96.68%)
client_sv2_setup_connection_serialize_deserialize	✅ (view plot)	35,582.00 (+0.09%)	35,362.69 (99.38%)	35,735.80 (99.57%)	✅ (view plot)	14,855.00 (+0.16%)	14,776.97 (99.47%)	14,886.07 (99.79%)	✅ (view plot)	21,822.00 (+0.17%)	21,695.27 (99.42%)	21,875.39 (99.76%)	✅ (view plot)	92.00 (-2.98%)	77.64 (84.39%)	112.02 (82.13%)	✅ (view plot)	380.00 (+0.08%)	375.40 (98.79%)	384.01 (98.96%)

Bencher - Continuous Benchmarking
View Public Perf Page
Docs | Repo | Chat | Help

roles/tests-integration/Cargo.toml

roles/Cargo.toml

roles/tests-integration/src/bin/run.rs

jbesraa · 2024-09-04T12:41:22Z

roles/Cargo.toml

@@ -23,6 +23,7 @@ members = [
    "translator",
    "jd-client",
    "jd-server",
+    "tests-integration",


maybe there is a better name.. roles-integration is other option

might be misleading? or can be roles-integration-tests

Yeah I also prefer roles-integration-tests. Also in the relative Cargo.toml

I think it should just be tests.

initially tests-integration was chosen because there is already a tests-utils folder and I wanted to be consistent. Is it worth deciding on the folder naming now, or when we know better how the role's crates are going to be structured/named? I prefer to keep it as is and rename it when restructuring the roles crates is underway, otherwise happy to change to either tests or roles-integration-tests, whatever gets more thumbs up.

makes sense!!!

rrybarczyk

This setup does not follow Rust's way of doing integration tests. See my comments on #1120:

Internal to crate integration test setup comment.
Multiple crate integration test setup comment.

If the intention of this PR is to setup the basic scaffolding of these tests, we should follow the aforementioned patterns.

jbesraa · 2024-09-09T16:14:40Z

This setup does not follow Rust's way of doing integration tests. See my comments on #1120:
* Internal to crate integration test setup [comment](https://github.com/stratum-mining/stratum/issues/1120#issuecomment-2338453729).

* Multiple crate integration test setup [comment](https://github.com/stratum-mining/stratum/issues/1120#issuecomment-2338487323).
If the intention of this PR is to setup the basic scaffolding of these tests, we should follow the aforementioned patterns.

Thanks for the review!

Do you have an example repo with the mentioned structure for 'Multiple crate integration test setup'?
From what I understand, a new crate is required in the workspace Cargo.toml and then the tests folder should live within this new crate.
If a new tests folder is created without a crate, whats the way to invoke the tests inside this folder and how crates are imported into the tests(internal or external)?
An example from tokio https://github.com/tokio-rs/tokio/tree/master/tests-integration

plebhash · 2024-09-09T20:26:07Z

This setup does not follow Rust's way of doing integration tests. See my comments on #1120:

Internal to crate integration test setup comment.

Multiple crate integration test setup comment.

If the intention of this PR is to setup the basic scaffolding of these tests, we should follow the aforementioned patterns.

I agree with the sentiment against adding yet another crate to the codebase.

However, given the fact that all roles are already separate crates under the same workspace, I'm afraid this is the only available path forward.

If we just added a tests/integration_test.rs file under the roles/Cargo.toml (which is a virtual manifest), we would be violating the fact that virtual manifests are not allowed to have bin or lib sections.

I tried implementing this locally and got stuck with:

error: failed to parse manifest at `/Users/plebhash/develop/stratum/roles/Cargo.toml`

Caused by:
  this virtual manifest specifies a `lib` section, which is not allowed

error: failed to parse manifest at `/Users/plebhash/develop/stratum/roles/Cargo.toml`

Caused by:
  this virtual manifest specifies a `bin` section, which is not allowed

Maybe in the future when we have a unified crate containing all roles (which is not something we're aiming for now), it will be easier to bypass the need for a new crate and we will be able to have all integration tests as part of the same crate (much like ldk-node).

For now, I could live with this tradeoff of having one extra crate inside roles, which seems hard to get around.

rrybarczyk · 2024-09-09T20:40:53Z

Ahh that is too bad.

Let's do what ldk-node does then.

rrybarczyk · 2024-09-09T20:42:50Z

I think we should consider having integration tests in each roles sub-crate though, wherever possible. Maybe post-refactor planning we can see what kind of things we can cover there.

jbesraa · 2024-09-11T07:21:15Z

I think we should consider having integration tests in each roles sub-crate though, wherever possible. Maybe post-refactor planning we can see what kind of things we can cover there.

Yup that would be great to have. But as mentioned, its really outside of the scope of this Pull Request.

Adds an initial integration tests project to the `roles` workspace. This crate should hold tests related to the other crates in the `roles` workspace, allowing us to write integration tests for those.

github-actions · 2024-09-11T07:43:09Z

Bencher

Report	Wed, September 11, 2024 at 08:24:10 UTC
Project	Stratum V2 (SRI)
Branch	1155/merge
Testbed	sv2

🚨 14 ALERTS: Threshold Boundary Limits exceeded!

Benchmark	Measure (units)	View	Value	Upper Boundary
client_sv2_handle_message_common	Instructions (instructions)	🚨 (view plot \| view alert)	501.00 (+6.11%)	492.99 (101.63%)
client_sv2_handle_message_common	L1 Accesses (accesses)	🚨 (view plot \| view alert)	779.00 (+6.09%)	765.39 (101.78%)
client_sv2_mining_message_submit_standard	Instructions (instructions)	🚨 (view plot \| view alert)	1,778.00 (+1.57%)	1,770.31 (100.43%)
client_sv2_mining_message_submit_standard	L1 Accesses (accesses)	🚨 (view plot \| view alert)	2,594.00 (+1.57%)	2,583.93 (100.39%)
client_sv2_mining_message_submit_standard_serialize	Instructions (instructions)	🚨 (view plot \| view alert)	4,722.00 (+0.59%)	4,714.31 (100.16%)
client_sv2_mining_message_submit_standard_serialize	L1 Accesses (accesses)	🚨 (view plot \| view alert)	6,806.00 (+0.72%)	6,788.09 (100.26%)
client_sv2_open_channel	Instructions (instructions)	🚨 (view plot \| view alert)	1,489.00 (+1.93%)	1,480.26 (100.59%)
client_sv2_open_channel	L1 Accesses (accesses)	🚨 (view plot \| view alert)	2,205.00 (+2.20%)	2,189.66 (100.70%)
client_sv2_open_channel_serialize	Instructions (instructions)	🚨 (view plot \| view alert)	5,092.00 (+0.56%)	5,083.26 (100.17%)
client_sv2_open_channel_serialize	L1 Accesses (accesses)	🚨 (view plot \| view alert)	7,377.00 (+0.73%)	7,357.90 (100.26%)
client_sv2_setup_connection	Instructions (instructions)	🚨 (view plot \| view alert)	1,530.00 (+1.88%)	1,521.26 (100.57%)
client_sv2_setup_connection	L1 Accesses (accesses)	🚨 (view plot \| view alert)	2,323.00 (+1.98%)	2,308.55 (100.63%)
client_sv2_setup_connection_serialize	Instructions (instructions)	🚨 (view plot \| view alert)	5,991.00 (+0.47%)	5,982.26 (100.15%)
client_sv2_setup_connection_serialize	L1 Accesses (accesses)	🚨 (view plot \| view alert)	8,712.00 (+0.59%)	8,696.48 (100.18%)

Click to view all benchmark results

Benchmark	Estimated Cycles	Estimated Cycles Results estimated cycles \| (Δ%)	Estimated Cycles Upper Boundary estimated cycles \| (%)	Instructions	Instructions Results instructions \| (Δ%)	Instructions Upper Boundary instructions \| (%)	L1 Accesses	L1 Accesses Results accesses \| (Δ%)	L1 Accesses Upper Boundary accesses \| (%)	L2 Accesses	L2 Accesses Results accesses \| (Δ%)	L2 Accesses Upper Boundary accesses \| (%)	RAM Accesses	RAM Accesses Results accesses \| (Δ%)	RAM Accesses Upper Boundary accesses \| (%)
client_sv2_handle_message_common	✅ (view plot)	2,119.00 (+2.08%)	2,206.38 (96.04%)	🚨 (view plot \| view alert)	501.00 (+6.11%)	492.99 (101.63%)	🚨 (view plot \| view alert)	779.00 (+6.09%)	765.39 (101.78%)	✅ (view plot)	2.00 (-66.16%)	14.47 (13.82%)	✅ (view plot)	38.00 (+1.37%)	41.12 (92.41%)
client_sv2_handle_message_mining	✅ (view plot)	8,188.00 (-0.15%)	8,353.00 (98.02%)	✅ (view plot)	2,137.00 (+0.17%)	2,180.60 (98.00%)	✅ (view plot)	3,163.00 (+0.28%)	3,230.38 (97.91%)	✅ (view plot)	32.00 (-12.67%)	45.52 (70.30%)	✅ (view plot)	139.00 (+0.05%)	142.39 (97.62%)
client_sv2_mining_message_submit_standard	✅ (view plot)	6,279.00 (+0.01%)	6,417.73 (97.84%)	🚨 (view plot \| view alert)	1,778.00 (+1.57%)	1,770.31 (100.43%)	🚨 (view plot \| view alert)	2,594.00 (+1.57%)	2,583.93 (100.39%)	✅ (view plot)	16.00 (-6.02%)	24.86 (64.36%)	✅ (view plot)	103.00 (-0.95%)	107.80 (95.54%)
client_sv2_mining_message_submit_standard_serialize	✅ (view plot)	14,626.00 (-0.81%)	15,059.49 (97.12%)	🚨 (view plot \| view alert)	4,722.00 (+0.59%)	4,714.31 (100.16%)	🚨 (view plot \| view alert)	6,806.00 (+0.72%)	6,788.09 (100.26%)	✅ (view plot)	38.00 (-14.40%)	55.36 (68.64%)	✅ (view plot)	218.00 (-1.75%)	230.61 (94.53%)
client_sv2_mining_message_submit_standard_serialize_deserialize	✅ (view plot)	27,458.00 (-0.16%)	27,895.83 (98.43%)	✅ (view plot)	10,613.00 (+0.44%)	10,640.30 (99.74%)	✅ (view plot)	15,448.00 (+0.47%)	15,483.64 (99.77%)	✅ (view plot)	78.00 (-4.58%)	90.51 (86.18%)	✅ (view plot)	332.00 (-0.83%)	346.00 (95.95%)
client_sv2_open_channel	✅ (view plot)	4,335.00 (-1.97%)	4,672.79 (92.77%)	🚨 (view plot \| view alert)	1,489.00 (+1.93%)	1,480.26 (100.59%)	🚨 (view plot \| view alert)	2,205.00 (+2.20%)	2,189.66 (100.70%)	✅ (view plot)	6.00 (-38.65%)	18.60 (32.26%)	✅ (view plot)	60.00 (-5.23%)	69.99 (85.73%)
client_sv2_open_channel_serialize	✅ (view plot)	13,922.00 (-1.32%)	14,533.56 (95.79%)	🚨 (view plot \| view alert)	5,092.00 (+0.56%)	5,083.26 (100.17%)	🚨 (view plot \| view alert)	7,377.00 (+0.73%)	7,357.90 (100.26%)	✅ (view plot)	28.00 (-20.66%)	44.01 (63.62%)	✅ (view plot)	183.00 (-3.09%)	200.92 (91.08%)
client_sv2_open_channel_serialize_deserialize	✅ (view plot)	22,621.00 (-0.07%)	23,036.45 (98.20%)	✅ (view plot)	8,055.00 (+0.59%)	8,080.53 (99.68%)	✅ (view plot)	11,721.00 (+0.61%)	11,757.82 (99.69%)	✅ (view plot)	73.00 (-0.83%)	85.13 (85.76%)	✅ (view plot)	301.00 (-0.79%)	314.97 (95.57%)
client_sv2_setup_connection	✅ (view plot)	4,693.00 (+0.12%)	4,784.27 (98.09%)	🚨 (view plot \| view alert)	1,530.00 (+1.88%)	1,521.26 (100.57%)	🚨 (view plot \| view alert)	2,323.00 (+1.98%)	2,308.55 (100.63%)	✅ (view plot)	5.00 (-45.33%)	15.86 (31.53%)	✅ (view plot)	67.00 (-0.79%)	70.28 (95.34%)
client_sv2_setup_connection_serialize	✅ (view plot)	16,092.00 (-0.65%)	16,516.88 (97.43%)	🚨 (view plot \| view alert)	5,991.00 (+0.47%)	5,982.26 (100.15%)	🚨 (view plot \| view alert)	8,712.00 (+0.59%)	8,696.48 (100.18%)	✅ (view plot)	34.00 (-17.81%)	55.11 (61.70%)	✅ (view plot)	206.00 (-1.64%)	217.94 (94.52%)
client_sv2_setup_connection_serialize_deserialize	✅ (view plot)	35,486.00 (-0.17%)	35,783.47 (99.17%)	✅ (view plot)	14,883.00 (+0.32%)	14,909.58 (99.82%)	✅ (view plot)	21,866.00 (+0.34%)	21,913.16 (99.78%)	✅ (view plot)	92.00 (-2.52%)	116.18 (79.19%)	✅ (view plot)	376.00 (-0.93%)	385.19 (97.61%)

Bencher - Continuous Benchmarking
View Public Perf Page
Docs | Repo | Chat | Help

github-actions · 2024-09-11T07:43:19Z

Bencher

Report	Wed, September 11, 2024 at 08:24:12 UTC
Project	Stratum V2 (SRI)
Branch	1155/merge
Testbed	sv1

Click to view all benchmark results

Benchmark	Estimated Cycles	Estimated Cycles Results estimated cycles \| (Δ%)	Estimated Cycles Upper Boundary estimated cycles \| (%)	Instructions	Instructions Results instructions \| (Δ%)	Instructions Upper Boundary instructions \| (%)	L1 Accesses	L1 Accesses Results accesses \| (Δ%)	L1 Accesses Upper Boundary accesses \| (%)	L2 Accesses	L2 Accesses Results accesses \| (Δ%)	L2 Accesses Upper Boundary accesses \| (%)	RAM Accesses	RAM Accesses Results accesses \| (Δ%)	RAM Accesses Upper Boundary accesses \| (%)
get_authorize	✅ (view plot)	8,458.00 (+0.15%)	8,794.94 (96.17%)	✅ (view plot)	3,746.00 (-0.05%)	3,882.19 (96.49%)	✅ (view plot)	5,253.00 (-0.07%)	5,440.98 (96.55%)	✅ (view plot)	4.00 (-42.99%)	11.64 (34.37%)	✅ (view plot)	91.00 (+0.99%)	95.35 (95.44%)
get_submit	✅ (view plot)	95,371.00 (-0.11%)	96,204.38 (99.13%)	✅ (view plot)	59,439.00 (-0.05%)	59,797.58 (99.40%)	✅ (view plot)	85,371.00 (-0.05%)	85,876.18 (99.41%)	✅ (view plot)	40.00 (-16.49%)	71.78 (55.73%)	✅ (view plot)	280.00 (-0.27%)	290.02 (96.54%)
get_subscribe	✅ (view plot)	8,059.00 (+0.63%)	8,361.53 (96.38%)	✅ (view plot)	2,841.00 (+0.16%)	2,964.68 (95.83%)	✅ (view plot)	3,969.00 (+0.10%)	4,136.82 (95.94%)	✅ (view plot)	13.00 (+0.24%)	23.26 (55.88%)	✅ (view plot)	115.00 (+1.17%)	119.63 (96.13%)
serialize_authorize	✅ (view plot)	12,301.00 (+0.38%)	12,636.48 (97.35%)	✅ (view plot)	5,317.00 (-0.04%)	5,453.19 (97.50%)	✅ (view plot)	7,416.00 (-0.04%)	7,603.29 (97.54%)	✅ (view plot)	4.00 (-56.70%)	15.05 (26.58%)	✅ (view plot)	139.00 (+1.57%)	143.70 (96.73%)
serialize_deserialize_authorize	✅ (view plot)	24,791.00 (+0.75%)	25,134.67 (98.63%)	✅ (view plot)	9,868.00 (-0.28%)	10,054.01 (98.15%)	✅ (view plot)	13,931.00 (-0.26%)	14,198.72 (98.11%)	✅ (view plot)	30.00 (-14.75%)	44.49 (67.43%)	✅ (view plot)	306.00 (+2.35%)	313.03 (97.75%)
serialize_deserialize_handle_authorize	✅ (view plot)	30,432.00 (+0.57%)	30,712.99 (99.09%)	✅ (view plot)	12,071.00 (-0.15%)	12,228.04 (98.72%)	✅ (view plot)	17,092.00 (-0.15%)	17,309.40 (98.74%)	✅ (view plot)	57.00 (+1.33%)	68.67 (83.01%)	✅ (view plot)	373.00 (+1.52%)	380.43 (98.05%)
serialize_deserialize_handle_submit	✅ (view plot)	126,558.00 (+0.08%)	127,172.41 (99.52%)	✅ (view plot)	73,280.00 (+0.01%)	73,673.76 (99.47%)	✅ (view plot)	105,058.00 (+0.02%)	105,619.15 (99.47%)	✅ (view plot)	100.00 (-9.37%)	142.54 (70.15%)	✅ (view plot)	600.00 (+0.66%)	604.70 (99.22%)
serialize_deserialize_handle_subscribe	✅ (view plot)	28,080.00 (+1.35%)	28,570.41 (98.28%)	✅ (view plot)	9,659.00 (+0.14%)	9,779.88 (98.76%)	✅ (view plot)	13,660.00 (+0.12%)	13,830.45 (98.77%)	✅ (view plot)	63.00 (-1.34%)	75.84 (83.07%)	✅ (view plot)	403.00 (+2.64%)	415.64 (96.96%)
serialize_deserialize_submit	✅ (view plot)	115,275.00 (+0.08%)	115,890.73 (99.47%)	✅ (view plot)	68,057.00 (-0.04%)	68,473.61 (99.39%)	✅ (view plot)	97,655.00 (-0.05%)	98,284.37 (99.36%)	✅ (view plot)	59.00 (-7.90%)	81.97 (71.98%)	✅ (view plot)	495.00 (+0.97%)	499.07 (99.18%)
serialize_deserialize_subscribe	✅ (view plot)	23,503.00 (+1.62%)	23,983.66 (98.00%)	✅ (view plot)	8,211.00 (+0.13%)	8,335.38 (98.51%)	✅ (view plot)	11,563.00 (+0.11%)	11,737.15 (98.52%)	✅ (view plot)	36.00 (-4.99%)	45.20 (79.64%)	✅ (view plot)	336.00 (+3.26%)	347.48 (96.70%)
serialize_submit	✅ (view plot)	99,836.00 (-0.04%)	100,546.19 (99.29%)	✅ (view plot)	61,483.00 (-0.05%)	61,846.33 (99.41%)	✅ (view plot)	88,211.00 (-0.05%)	88,728.35 (99.42%)	✅ (view plot)	43.00 (-12.42%)	70.80 (60.73%)	✅ (view plot)	326.00 (+0.31%)	333.36 (97.79%)
serialize_subscribe	✅ (view plot)	11,456.00 (+0.62%)	11,761.89 (97.40%)	✅ (view plot)	4,188.00 (+0.11%)	4,311.68 (97.13%)	✅ (view plot)	5,826.00 (+0.07%)	5,993.78 (97.20%)	✅ (view plot)	13.00 (-5.26%)	22.28 (58.36%)	✅ (view plot)	159.00 (+1.28%)	164.54 (96.63%)

Bencher - Continuous Benchmarking
View Public Perf Page
Docs | Repo | Chat | Help

github-actions · 2024-09-11T07:43:25Z

Bencher

Report	Wed, September 11, 2024 at 08:23:52 UTC
Project	Stratum V2 (SRI)
Branch	1155/merge
Testbed	sv2

Click to view all benchmark results

Benchmark	Latency	Latency Results nanoseconds (ns) \| (Δ%)	Latency Upper Boundary nanoseconds (ns) \| (%)
client_sv2_handle_message_common	✅ (view plot)	44.16 (-1.20%)	46.72 (94.54%)
client_sv2_handle_message_mining	✅ (view plot)	79.63 (+5.93%)	91.81 (86.73%)
client_sv2_mining_message_submit_standard	✅ (view plot)	14.66 (-0.11%)	14.81 (98.96%)
client_sv2_mining_message_submit_standard_serialize	✅ (view plot)	264.06 (-1.49%)	291.43 (90.61%)
client_sv2_mining_message_submit_standard_serialize_deserialize	✅ (view plot)	590.83 (-0.33%)	650.41 (90.84%)
client_sv2_open_channel	✅ (view plot)	165.77 (-0.08%)	177.56 (93.36%)
client_sv2_open_channel_serialize	✅ (view plot)	276.95 (-0.05%)	307.30 (90.12%)
client_sv2_open_channel_serialize_deserialize	✅ (view plot)	373.46 (-1.07%)	458.97 (81.37%)
client_sv2_setup_connection	✅ (view plot)	162.05 (+0.04%)	185.93 (87.16%)
client_sv2_setup_connection_serialize	✅ (view plot)	471.41 (+3.05%)	568.83 (82.87%)
client_sv2_setup_connection_serialize_deserialize	✅ (view plot)	936.88 (-6.10%)	1,163.53 (80.52%)

Bencher - Continuous Benchmarking
View Public Perf Page
Docs | Repo | Chat | Help

rrybarczyk

Lgtm

Shourya742

ACK

jbesraa marked this pull request as ready for review September 4, 2024 09:50

jbesraa force-pushed the 2024-09-04-int-test branch 2 times, most recently from bd8390d to fb554bc Compare September 4, 2024 10:15

This was referenced Sep 4, 2024

Add Template Provider to integration test project #1156

Merged

PoolSv2 integration tests #1066

Merged

jbesraa commented Sep 4, 2024

View reviewed changes

roles/tests-integration/Cargo.toml Outdated Show resolved Hide resolved

GitGab19 reviewed Sep 4, 2024

View reviewed changes

roles/Cargo.toml Outdated Show resolved Hide resolved

roles/tests-integration/src/bin/run.rs Outdated Show resolved Hide resolved

jbesraa force-pushed the 2024-09-04-int-test branch from fb554bc to d5bd407 Compare September 4, 2024 12:35

jbesraa commented Sep 4, 2024

View reviewed changes

plebhash self-requested a review September 4, 2024 20:11

rrybarczyk requested changes Sep 9, 2024

View reviewed changes

jbesraa force-pushed the 2024-09-04-int-test branch 2 times, most recently from 863cf29 to 6eb932a Compare September 11, 2024 07:25

Add integration tests project

633684a

Adds an initial integration tests project to the `roles` workspace. This crate should hold tests related to the other crates in the `roles` workspace, allowing us to write integration tests for those.

jbesraa force-pushed the 2024-09-04-int-test branch from 6eb932a to 633684a Compare September 11, 2024 07:34

jbesraa mentioned this pull request Sep 12, 2024

Add integration test for auth failure between mining_device and a pool #1164

Closed

rrybarczyk approved these changes Sep 13, 2024

View reviewed changes

Shourya742 approved these changes Sep 14, 2024

View reviewed changes

plebhash merged commit e55cf5a into stratum-mining:main Sep 15, 2024
34 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add integration tests project #1155

Add integration tests project #1155

jbesraa commented Sep 4, 2024

github-actions bot commented Sep 4, 2024 •

edited

Loading

github-actions bot commented Sep 4, 2024 •

edited

Loading

github-actions bot commented Sep 4, 2024 •

edited

Loading

github-actions bot commented Sep 4, 2024 •

edited

Loading

jbesraa Sep 4, 2024

Shourya742 Sep 4, 2024

GitGab19 Sep 5, 2024

rrybarczyk Sep 9, 2024

jbesraa Sep 11, 2024

Shourya742 Sep 14, 2024

rrybarczyk left a comment

jbesraa commented Sep 9, 2024

plebhash commented Sep 9, 2024

rrybarczyk commented Sep 9, 2024

rrybarczyk commented Sep 9, 2024

jbesraa commented Sep 11, 2024

github-actions bot commented Sep 11, 2024 •

edited

Loading

github-actions bot commented Sep 11, 2024 •

edited

Loading

github-actions bot commented Sep 11, 2024 •

edited

Loading

rrybarczyk left a comment

Shourya742 left a comment

Add integration tests project #1155

Add integration tests project #1155

Conversation

jbesraa commented Sep 4, 2024

github-actions bot commented Sep 4, 2024 • edited Loading

Bencher

github-actions bot commented Sep 4, 2024 • edited Loading

Bencher

github-actions bot commented Sep 4, 2024 • edited Loading

Bencher

github-actions bot commented Sep 4, 2024 • edited Loading

Bencher

jbesraa Sep 4, 2024

Choose a reason for hiding this comment

Shourya742 Sep 4, 2024

Choose a reason for hiding this comment

GitGab19 Sep 5, 2024

Choose a reason for hiding this comment

rrybarczyk Sep 9, 2024

Choose a reason for hiding this comment

jbesraa Sep 11, 2024

Choose a reason for hiding this comment

Shourya742 Sep 14, 2024

Choose a reason for hiding this comment

rrybarczyk left a comment

Choose a reason for hiding this comment

jbesraa commented Sep 9, 2024

plebhash commented Sep 9, 2024

rrybarczyk commented Sep 9, 2024

rrybarczyk commented Sep 9, 2024

jbesraa commented Sep 11, 2024

github-actions bot commented Sep 11, 2024 • edited Loading

Bencher

github-actions bot commented Sep 11, 2024 • edited Loading

Bencher

github-actions bot commented Sep 11, 2024 • edited Loading

Bencher

rrybarczyk left a comment

Choose a reason for hiding this comment

Shourya742 left a comment

Choose a reason for hiding this comment

github-actions bot commented Sep 4, 2024 •

edited

Loading

github-actions bot commented Sep 4, 2024 •

edited

Loading

github-actions bot commented Sep 4, 2024 •

edited

Loading

github-actions bot commented Sep 4, 2024 •

edited

Loading

github-actions bot commented Sep 11, 2024 •

edited

Loading

github-actions bot commented Sep 11, 2024 •

edited

Loading

github-actions bot commented Sep 11, 2024 •

edited

Loading